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Diversity in Program Improvement Approaches: 
Inplications for Technical Assistance 



I. INTRODUCTION 



The Title 1/Chapter 1 Technical J\ssistance Centers (TACs) were 
established in 19-^6 by the U.S. Office of Education (now U.S. Department 
of Education) to provide evaluation technical assistance to state 
education agencies (SEAs) and local educrtion agencies (LEAs) . Since 
their inception, the TACs have been responsible for helping client 
agencies to iirprove their evaluation capabilities, to promote the use of 
evaluations, and to embark on program improvement activities (Millraan, et 
al., 1979; Reisner, et al. , 1982). The overall mission has remained 
unchanged over the past nine years although increasingly greater emphasis 
is now placed on program improvement. 

The 1984-85 contract year began with a directive from the U.S. 
Department of Education that ax. lear^t 25 percent of TAG resources be 
expended on program improvement activities. The new funding cycle, 
beginning in 1985-86, required the TACs to devote up to one-half ot their 
resources to assisting SEAs and LEAs in program improvement. In 
addition, the Secretary's Initiative to improve Ch^ter 1 proj3cts and to 
recognize unusually effective Chapter 1 projects provided another 
integral part of the overall TAC effort to help improve instructional 
services provided to disadvantaged youngsters. 

In carrying out program improvement work, the Region 4 TAC, which 
serves 13 western states, the Pacific Trust Territories and the Bureau of 



Indian Affairs, has relied heavily on available literature and research 
findings relating to effective schools. Many the findings have been 
translated into ideas which can be put into practice in the day-to-day 
school setting. 

A variety of approaches has been used to initiate program improvement 
activities at the SEA and LEA levels. The adoption of a particular 
approach is otten determined by contextual factors unique to an SEA or 
LEA setting. The approach is often influenced and sh^ed by ^ecific 
needs of clients, their perspectives on the relationships between the 
help giver (TAG) and the recipient (clients) , and the essential 
ingredients of change in education. 

The impetus for change may be exogenous (e.g., a Congressional 
mandate, a directive from the U.S. Department of Education, TAG 
recommendations) or endogenous (e.g., an intrinsic desire to do better). 
Likewise, the change agent can be external (e.g., TAG staff) or internal 
(e.g., SEA or LEA personnel). Strong commitment on the part of the help 
recipient is critical to the success of any change effort. It is also 
essential that the change activities produce some discernible outcomes to 
maintain the momentum and to ultimately lead to the incorporation or 
institutionalization of the cuange effort. 

Program improvement activities can be classified into four categories 
on the basis of the source of the impetus (exogenous versus endogenous) 
and che locus of the change agent (external versus internal). A 
systematic improvement effort occurs when the impetus is endogenous and 
the change agent is internal. Gonversely, a symbolic improvement effort 
takes place when the inpetus is exogenous and the change agent is 
external. An opportunistic improvement activity is implemented with an 
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endogenous impetus and an external change agent, A pragmatic improvement 
activity is iirpleraented witn an exogenous impetus and an internal change 
agent* While the four ti ^es of program improvement activity are not 
equally desirable, they each have their strengths and weaknesses when 
initiated in real life school settings. 

The objectives of this paper are to (a) describe the various 
approaches to implementing Chapter 1 program improvement activities at 
the state and local district levels in 13 western states through the 
provisiixi of technical assistance by an outside agency; (b) categorize 
these approaches in terms of essential ingredients for fostering change 
in education; and (c) discuss implications of the findings for future 
work in program improvement* 



II • METHOD 

The study is based on data gathered through onsite contact logs and 
monthly reports submitted to the U.S. Department of Education for the 
1982-85 contract years (NWREL, 1985). Data elements contained in these 
documents included: 



o Number of onsite activities (e.g., workshops and consultations 

with TAG clients) 
o Topics addressed in each onsite activity 
o Number of clients served in each onsite activity 
o Length of each onsite activity in hours 



These data were summarized by month and then by each of the contract 



years. For purposes ot the present study, data for all three contract 
years were combined to obtain an aggregate picture for che entire 
contract period. The numerical data were augmented by field notes which 
provided anecdotes and vignettes of program improvement activities at 
both the SEA and LEA levels. 

The program improvement activities were then categorized according to 
the conceptualization described earlier. While the categories are by no 
means exclusive of each other, each program improvement effort was 
categorized as systematic, opportunitistic, pragmatic, or symbolic. 



III. FINDINGS 

The data showed that during the 1982-85 contract years, the Region 4 
TAG provided 1,976 workshops and onsite consultations to a total of 25, 
933 SEA and LEA personnel. These onsite activities dealt with over 50 
different topics relating to Ch^ter 1 program operations and outcomes. 
Increasingly, the most requested topics included those pertaining to 
program improvement* Specifically, over 300 workshops and consultations 
were provided on program iraproveraent topics. 

Opportunism 

OE^>ortunistic program improvement activities were initiated by TAG 
clients, but the change agent role was played largely by TAG staff. This 
often occurred when the client agency found itself lacking sufficient 
resources and/or expertise to carry out the change function. Examples of 



O 4 R 

ERIC 



these projects included: 

o Assessing the effects of a particular program coraponeat 

o Assessing and increasing time-on- task 

o Developing data systems to monitor student progress 

o Looking at parent involvement activities to determine their 

impact on student outcomes 
o Assessing the relative impact of service delivery on different 

types of students 

o Reviewing software packages for program use as part of program 
improvement 

Example; A Family Goal Program 

The Ch^ter 1 program on the Big Island of Hawaii has a unique parent 
involvement component called the Family Goal Program, The component 
encourages active participation of parents in the learning procer*' Each 
Chapter 1 parent enters into an agreement with the district to set a 
family goal and to engage in activities designed to achieve that goal. 
Sairple family goals include: 

o Set aside a quiet place to study and to do home work 

o Take our child to the library regularly 

o Read to our child 

o Talk to our child about books read 

o Turn off the TV at a specified time each day so the family can 
read together 
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o Ask the Chapter 1 teacher how we can heJp our child 
o Get more books for our home. 

A family goal log is provided for parents to keep track of activities 
and progress. The log is forwarded to the Chapter 1 teachers for 
examination and recordkeeping on a regular basis. 

Although the program coniponent is widely perceived to be a n ijor 
conf^ibuting factor tu the success of the Chapter 1 program, no 
systematic stcdy has been conducted to assess its effects on student 
performance. Thus, when the district submitted an application to the 
State Department of Education for recognition of the conponent as an 
exemplary practice, the state validation team recommended that the 
ccwoponent be evaluated for evidence of efficacy. 

The district staff, perceiving a need to demonstrate the effects of 
the component and to further strengthen the conponent, invited TAC s*-.aff 
to design a con?>rehensive improvement-oriented evaluation. The study is 
to provide strong evidence of efficacy (or the lack of it) and rich 
information for improvement. 

In early 1985, TAC developed a longitudinal design (3-5 years), using 
entering students as cohorts. Chapter 1 schools in the district were 
randomly designated as experimental or control schools after they had 
been paired off on the basis of school locale, enrollment and grade span 
coverage. Seven schools were designated as experimental schools. 
An >ther seven served as controls. 

The evaluation study will last five years (1985-86 through 1989--90) , 
unless unequivocal evidence is gathered during the first three years of 
the study. The design calls for increasing numbers of student cohorts in 
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progressive stages* In the first year, the first cx>horts will be in the 
study. The second year will include two categories of cohorts — those 
receiving the treatraent for the second year and those having their firs - 
exposure to the treatment. Progressively, each of the following years 
will have the benefit of additional cohorts. 

The dependent variables will include reading achievement gains in 
Normal Curve Equivalents (NCEs) , school grades, attendance, reading 
attitudes and parent perceptions. The independent variables will include 
nature of the family goals, number of goals, frequency of activity, grace 
level, school locale, school year and project setting. 

TAC stalf conducted two workshops in late 1985 as part of the 
evaluation process. In the first workshop, a tentative evaluation plan 
was presented to all district staff to familiarize them with the 
objectives ot the study and to solicit ir^ut for any necessary 
modifications. • The reception from the staff was highly favorable. The 
second workshop occurred after the evaluation plan was approved by the 
district superintendent. TAC staff e:q>lained the design and logistics 
to the Chapter 1 staff. The evaluation commenced with the 1985-86 school 
year. 

The evaluation instruments include the family goal log, school 
records, the Metropolitan Achievement Test, a reading attitude inventory 
and a parent survey questionnaire. The inventory and questionnaire were 
adapted from existing survey instruments ana field tested in the latter 
part of 1985 with small sanples of students and parents. Both 
instruments were substantially revised and improved. 

The five-year evaluation is expected to provide answers to the 
following questions: 



1. Does the family goal program component have a positive impact on 
Chapter 1 outcomes in student reading achievement, attendance, 
reading attitudes and parent perceptions? 

2. Do the numbar and nature of specific fairily goals have any 
effects on students outcomes? 

3. What is the relationship between the frequency of family goal 
activities and student outcomes? 

4. Do differential effects occur with reject to grade level, 
school locale and project setting? 

5. In what ^ecific ways can the family goal program component be 
improved? 

TAG staff will continue to work closely with the district staff in 
iii?>lementing tne evaluation study. We will use the evaluation 
irifonaation not only to assess the overall efficacy of the program 
conponent, but also to find ways of implementing improvement activities 
to further strengthen the program component. 



Pragmatism 

Pragmatic improvement activities were often initiated to address some 
aspects of program operations which had become mini-crises. They often 
called for trouble- shooting to put out "brush fires." These projects 
included : 

o Providing testwiseness training to teachers 
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o Uj gracing student selection procedures 
o Upgrading needs assessment procedures 

o Integrating Chapter 1 and regular program through coordinated 

needs assessment and process evaluation 
o Identifying classroom behaviors affecting student achievement 

and develc^ing curricular materials and activities to cultivate 

such behaviors 

o Ujing a program implementation checklist to assess status of 

program implementation and to identify highly effective program 
elements 

Exartg)le; Student Selection 

For years^ the Kauai district in Hawaii has used test scores as the 
sole criterion for student selection. There was a widespread concern 
among the staff that test data did not provide an adequate basis for 
identifying eligible students. The process allowed too many students 
deserving help to "fall through the cracks." 

In late 1984, at the request of the district, a TAG workshop was 
provided to explore various options for student selection, including the 
use of multiple measures and composite scoces. Following the workshop, 
the district Ch^ter 1 resource teacher, working with the Chapter 1 
staff, developed a process for student selection which involved the use 
of non-test indicators. These indicators included student behavioral 
characteristics, teacher ratings on specific reading problems, and other 
cognitive and affective factors such as school grades, participation in 
other qpecial program, self-concept and work habits. 
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A checklist was created to obtain the necessary student information. 
Chapter 1 teachers worked collaboratively with the regular teachers to 
complete a checklist for each student scoring below the 30th percentile 
on a norm- referenced test. The various measures were weighted 
differentially and combined into a composite score for student selection* 

In early 1985 r the process was piloted with a sanple of students. 
Changes were made in the contents of the checklist as well as the 
procedures for deriving the conrposite score. In late 1985^ following the 
use of the new process for a school year^ the process was computerized to 
eliminate the need for time-consuming calculations by teachers. 

The Chapter 1 teacners generally felt thirt the new process was 
superior to the old one and were highly appre -jiative of the fact that the 
new process proviaes a built-in mechanism for coordinating wich the 
regular teachers in providing instruction to Chapter 1 youngsters. 



Systematic improvement activities addressed lOiiger-terra goals and the 
change process could readily be incorporated in the existing system. 
While TAC staff might play a crucial role in such efforts, both the 
inpetus and the change agent resided within the existing SEA or LEA 
system. Exan?>les of such projects included: 

o Using the SEA monitoring process as a means of program 



Systematism 



improvement 



o 



Using the LEA monitoring process as a means of program 



improvement 
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o Developing long-range improvement plans and activities involving 
the establishment of an iiiternal leadership team in a local 
setting 

Examp3e: Mcxiitoring by State Education Agency 

In carrying out their monitoring function in Ch^ter 1, man^ state 
ed»ication agencies concentrate on program caiplian'?'^ issues relating to 
stcite and lederal rules and regulations, Littl ttention is paid to 
providing guidance and information for in-depth program improvement. 
State level personnel are seen aa compliance enforcement agents rather 
than technical assistance providers. 

To alter that image, the Hawaii Department of Education started a 
program in 1984 to promote program improvement activities at the local 
school level. Spearheaded by ti^e Compensatory Kducation Section, the new 
procedures provided guidance and information to project staff and 
adirinistrators for program planning and improvement. In return, each 
project school documented program improvement activities on a continual 
basis, and shared the information with other project schools. 

The effort, partly funded tnrough a grant from the U.S. Department of 
Education, included a staff development component in which the state 
monitoring personnel received training from TAG starf in daua collection 
strategies and the dissenination of project improvement information. 

A project review committee was formed to review and react to all 
processes, activities, and materials developed in the project. The 
committee represented a cross section of the d^artment work force and 
e:^.3rtise, including state, district, and school level staff and a TAG 
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staff. Based on the most recent research on effective schools, the 
committee identified and operationalized variables contributing to high 
student achievement. These variables included engaged time, 
student- teacher interaction, instructional strategies, and student 
motivation. With assistance from TAG staff, the committee developed, 
adopted, or adapted instruments to gather the relevant information. TAG 
staff then trained state and district staff in the use of the 
instruments, as well as in data analysis and interpretation. 

The process included intensive classroom observations to generate 
information for program improvement. In addition, the program 
improvemeiit team interviewed school administrators to gather information 
on their expectations, leadership, and support of particular program 
activities. Following the observations and interviews, the team provided 
immediate feedback to principals and Ghapter 1 instructional staff. 

A pilot test of the new procedures took place in the tarly part of 
"^984 in six elementary schools in the Maui School District. To assess 
the injjact of the p^- process, a questionnaire was created and 
administered first in the fall of 1983, and then in the spring of 1984, 
to all Ghapter 1 instructicxial staff in the six schools. In addition, 
the proj«^ct staff used a program improvement log to document all 
improvement activities between November 1983 and April 1984. 
The questionnaire surveys provided the following findings: 

1# There were favorable differences between pre and post 

administrations of the questionnaire, with rei^ect t * the 
respondents' familiarity with the purpose and procedures of 
state monitoring. As would be expected, the respondents. ,^ere 
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more familiar with the process at post administration than they 
were at pre administration. 
2» At post administration, the respondents were more positive with 
req>ect to the amount of program improvement information 
provided iDy the monitoring process than they were at pre 
admin istration. 

3. The discrepancies between what the respondents felt the 

monitoring process should do and v^at it actually did in terms 
of providing program improvement information were narrowed 
considerably between pre and post administrations. 

The program improvement log recorded a total of 51 improvement 
activities between November 1983 and April 198--. Program improvement 
areas included parent involvement, professional improvement, student 
motivation, program evaluation, rlassroom management, and instructional 
strategies. 

Outcomes of the study suggest that a state education agency can 
attain a high degree of success in revamping its monitoring function from 
compliance to program improvement. The study changed the perceptions of 
local project staff regarding program improvement and the state 
monitoring process. Project staff began to look to or actually use the 
state monitoring process as a significant source ot program improvement 
information. 



13 



15 



Example; Quality Monitoring by Local Education Agency 



Quality monitoring as in5)lemented in the Honolulu district in Hawaii 
is a participatory and collaborative process to bring about positive 
changes in Chc5)ter 1 project schools. The process examines project 
implementation variables and evaluation data and translates them into 
action plans to ensure fidelity of program implementation and to improve 
student achievement. An ^ii^rtant outcome of quality monitoring is the 
continuous exploration for better ways of providing instruction to 
Chapter 1 participants. 

The process begins when district staff examine achievement gains for 
each project school in conparison to previous years, to district 
averages, and to grade level averages. Significant patterns, if they 
exist, are detected and implementation variables are analyzed. The 
district staff then prepare the project staff and administrators for the 
initial quality monitoring meeting. 

As tlie district staff review project implementation and outcomes, 
they consider a host o£ variables such as classroom environment, 
instructional delivery systems, per pupil costs, evaluation procedures, 
evaluation results, test performance of students, instructional 
strategies, inservice ne^s of project staff, and parent involvement. 

At the initial quality monitoring meeting, district and school level 
personnel develop a plan of action and identify persons responsible for 
carrying out the plan. Decisionmaking and responsibility are shared by 
all individuals involved in the project. 

Since the quality monitoring process was f.rrst inpleraented in 1980, 
Ch^ter 1 projects selected to participate in the process have met with 
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much success. Achievement gains have invariably resulted from the total 
commitment of project staff to this process. Dole Intermediate School^ 
the third largest intermediate school in the district, typifies the 
success story. 

In examining the evaluation results for Dole Intermediate for the 
1980-81 and 1981-82 school years, district staff noted that Chapter 1 
students in grades eight and nine had not made the anticipated gains. 
The ninth graders, in particular, showed negative NCE gains for both 
school years. The decreasing gains prompted the selection of the school 
for quality monitoring in 1982-83. 

The initial quality monitoring meeting occurred at the beginning of 
the new school year in October. During the meeting, attention focused on 
the poor achievement of Chapter 1 participants in grades eight and nine. 
Variables t' mght to have some impact on the low achievement level were 
identified. The plan of action was to reassign a Chapter 1 teacher from 
grade seven to grade eight. Project students were reassigned so that the 
larger classes were reorganized into smaller classes. The school 
administrator, the dirtrict resource teacher, the school registrar, and 
the project teache • coordinator were all responsible for the 
reorganization which took place shortly after the quality monitoring 
meeting • 

Immediately following the initial quality monitoring meeting, the 
district resource teacher met with the project staff to review the plan 
of action and to work out further details. A detailed plan of action to 
effect changes wes drawn up and put in place. As other concerns arose 
during the school year, new plans were developed and implemented. On a 
continual basis, the project staff shared information, insights, and 
evaluations with the district resource teacher and other colleagues. 
O 15 1 
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At the end of the school year, the ninth graders showed an NCE gain 
of 5.1, as conpared with a negative gain of -1.7 the previous year. The 
school wide average increased from 2.5 to 5.2. The evaluation results 
were reviewed by district staff at the beginning of the following schrx^l 
year as a new quality monitoring cycle began. 

Symbolism 

Symbolic improvement activities were prompted by an outside impetus 
and involved an external change agent. These were often discrete events 
the occurrence of which was largely dependent on outside intervention. 
Examples of these activities included: 

o Participating in state identification, validation and 

dissemination process 
o Participating in the Joint Dissemination and Review Panel process 
o Participating in the National Identification Program for 

unusually effective Ch^ter 1 projects 



Example; The National Identification Program 

In November 1984, the U.S. Education Department initiated a program 
to identify Chc4>ter 1 projects that have been unusually successful in 
meeting the fecial needs of disadvantaged students. Referred to as the 
National Identification Program, the effort was part of the Secretary's 
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Initiative to improve Chapter 1 through the sharing of effective 
practices ^ecific to compensatory education settings. 

The program requested nominations from state education agencies in 
the 50 states, the District of Columbia, Puerto Rico and the Bureau of 
Indian Affairs. Each Chapter 1 project was asked to submit demographic 
data as well as information on 13 program attributes and four achievement 
indicators. The 13 attributes comprised indicators of effective programs 
roost often cited in the current school improvement research. . ejects 
were asked to highlight those attributes that were implemented in a 
unique manner and contributed most significantly to program effectiveness. 

The 13 program attributes were: 

o Clear project goals and objectives 

o Coordination with the regular school program and/or other 

fecial programs 

o Parent and community involvement 

o Professional development and training 

o Strong leadership 

o Appropriate instructional materials, methods and approaches 

o High expectations for student learning and behavior 

o Positive school and classroom climate 

o Maximum use of academic learning time 

o Closely monitored student progress 

o Regular feedback and reinforcement 

o Excellence recognized and rewarded 

o Evaluation results used for program improvement 

O 17 

ERIC i9 



By early 1985, the state education agencies submitted 333 nominations 
to the U*S# Education Department. The nominated projects covered a wide 
diversity of locations, settings, philosophies and instructional 
approaches. The submissions included regular Chapter 1 projects, migrant 
projects, as well as projects for neglected or delinquent youth 
populations. 

Each nomination was reviewed by a panel consisting of representatives 
from major educational associations, school improvement research and 
compensatory education. Panel members examined each nomination and 
pcq>ared a report, including a summary of the ratings and comments. 

The review panels identified 116 of the 333 projects submitted by 
states as unusually successful and most ready to be shared with other 
Chapter 1 projects. These projects covered a vast geographic spread of 
the country, including 23 p.:ojects from Region 4. On April 11, 1985, the 
U.S. Education Department officially designate^i the 116 projects as 
worthy of fecial recognition. 

In conjunction with the National Identification Program, a two-volume 
Chapter 1 Sourcebook has been prepared for national dissemination. 
Volume I presents findings from effective schools research as they relate 
to the 13 program attributes with special reference to compensatory 
education. Volume II provides descriptive information on the identified 
projects and how they have inplemented effective practices. In addition, 
all Title I and Chc4>ter 1 projects currently approved by the Department 
of Education's Joint Dissemination Review Panel (JDRP) are also included 
in the document. 

Regional conferences are currently being held for the identified 
projects to share information on their effective practices with other 
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Chapter 1 projects* Future TAG workshops on program improvement will 
also include the use of the Sourcebook as a rf^source for implementing 
iirqorovement activities* 



IV* ASSUMP^'IONS 

TAG assistance in program improvement, like most school improvement 
packages, resets on assumptions which often stay inplicit in the packaging 
process. To be successful, TAG staff need tc be mindful of these 
assumptions and to assess their validity in each improveinent effort. 

Assumption One ; With a minimum amount of technical assistance, 

existing Ghapter I staff are capable of learning from 
and applying research findings to program improvement 
efforts* 

Few ii^provement efforts have called for a substantial increase in 
project funding* Most improvement activities are carried out by existing 
staff, promotin<) a more efficient use of existing resources (Edmonds, 
1982)* Indeed, if eveiry improvement effort requires a change in 
personnel, very few efforts will be fiscally or politically feasible* 

Most improvement approaches establish new cooperative diagnostic and 
problem- solving groups to make things happen. These groups are variously 
called quality circles, quality of (school) life committees, 
problem- solving task forces or leadership teams (Pratzner, 1984)* Tliey 
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provide the opportunity for people to identify barriers to the 
effectiveness of their organization and, t'^ rough problem- solving, bieak 
down those barriers. The approach is based on the conviction that within 
the organization there exists a largely untapped pool of creative 
e:q?ertise and insight which the existing organizational structure has 
hidden or suppressed* This latent talent can be channeled and put to 
work to increase program effectiveness 

Assumption Two ; Past research has generated a knowledge base to inform 

program improvement efforts. 

In recent years, researchers have made a concerted effort to build 
and disseminate a knowledge base on effective schools. Edmonds (1982), 
for exanple, offers the following characteristics of an effective school 
(page 4) : 

1, The principal's leadership and attention to the quality of 
iPiStruction; 

2, a pervasive and broadly understood instructional focus; 

3, an orderly, safe climate conducive to teaching and learning; 

4, teacher behaviors that convey the expectation that all students 
are expected to obtain at least minimum mastery; and 

5, the use of measures of pupil achievement as the basis for 
program evaluation. 

Mackenzie (1983), in a perhaps more comprehensive review, identifies 
31 elements, clustered aiound the dimensions ot leadership, efficacy, and 
efficiency. The ••core" elements include (page 8) : 



1. Positive climate and overall atnxjsphere; 

2. goal focused activities toward clear, attainable and relevant 
obj ectives; 

3. teacher-directed classroom management and decLsion-making ; 

4. inservice staff training for effective teaching; 

5. high and positive achievement expectations with a constant press 
for excellence; 

6. visible rewards for academic excellence and growth; 

7. coc^erative activity and group interaction in the classroom? 

8. total staff involvement with school improvement; 

9. autonomy and flexibility to implement adaptive practices; 

10. appropriate levels of difficulty for learning tasks; 

11. teacher enpathy, rapport, and personal interaction with students; 

12. effective use of instructional time, amount and intensity of 
engagement in school learning; 

13. orderly and disciplined school and classroom environments; 

14. continuous diagnosis, evaluation and feedback; 

15. well-structured classroom activities; 

10. instruction guided by content coverage; and 

17 schoolwide emphasis on basic and higher order skills. 



The knowledge base tor effective schools draws from findings of 
several areas of research, including school effects, teacher effects, 
instructional leadership, curriculum alignment, organizationrl 
development and educational change. The broad knowledge base has served 
as a powerful iraoetus and a rich resource for a variety of school and 
program improvement projects. 

23 
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Assumption Three : Loosely coupled intra-school systems (e.g 



classrooips, programs, department^:} can be changed in;:o 



a coh3sive, purposeful organization by implementing 



control systems developed through a participatory 



process* 



Effective schools proponents advocate using schools as the unit of 
change. For exampl(», in a recent survey of 25 states. Miles, Farrar, and 
Neufeld (1&33) found that nearly all of the 39 effective school programs 
had a strong ^phasis on the school as an organization. They emphasized 
inq?rovement at the school level. Passalaqua (1981) similarly concluded 
that unless the school as a functioning social system is the focus of 
social change, program adoption and effective reform are not likely to 
occur. Using schools as the unit makes it possible to have multiple 
changes made simultaneously wh: ^h research on quality of work life has 
shown to be an effective way of improving coni>lex social systems 
(Pratzner, 1984) . 

However, research has also shown that schools are •'loosely coupled" 
systems in which teachers are largely independent of the principal's 
immediate supervision (Weick, 1976) . Indeed, Firestone & Herriott (x982) 
found that teachers have nore influence than principals over day-to-day 
management decisions. As a result, attempts to increase school 
effectiveness through imposing discrete policies by fiat are nv->t likely 
to be fruitful (Purkey & Smith, 1982). Ultimately, teachers have the 
power to make or break the improvement effort (Sirotnik, 1984). 

Successful innovations are often those that have been developea,. or 
at least significantly altered, by teachers who are expected to implement 
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the innovation (Bennan & McLaughlin^ 1974; Cooley, 1983). In a loosely 
coiflpled setting, a change effort is often (a) mutually adapted, (b) not 
inqpleraentedr or (c) coopted by the participants. 

It is obvious that instructional improvement would seldom occur 
without the encouragement and support of the principal. Research on 
effective instruction, by itself, often does not have sufficient appeal 
to galvanize teachers into changing their instructional practices. The 
principal must provide the needed impetus to stimulate faculties into 
action by fostering a positive climate for academic achievement, by 
hammering out long-range goals, and by supporting teacher efforts to 
reach those goals (Mackenzie, 1983) • 

It is also obvious, however, that the effects of a school's 
"syndrome," "culture," or "ethos," (Purkey & Smith, 1983), need to move 
from the school level to the individual classroom. Ultimately, the 
classroom is where learning takes place. School effectiveness, if it 
means anything, comes down to behavior change in the classroom 
(Tomlinson, 1981) . 

In the Chapter 1 National Identification Program for 1984, the U.S. 
Education Department identified 13 attributes of effective schools and 
classrooms. These attributes included instructional and organizational 
variables. For the 116 projects receiving national recognition, the 
predominant attribute was an effective instructional process. Over 65 
percent of the projects reported appropriate instructional materials and 
methods as the major reason for their success. Organizational variables 
such as school climate, goals and objectives, and leadership were cited 
by less than one-fourth of the projects as particularly salient factors. 
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Perhaps Pratzner (1984) strikes a desired balance when he says: 
"Institutional improvement must go 
Land- in- hand with individual improvement, 
and those who are closest to the work that 
needs to be performed (students and 
teachers) are also the most knowledgeable 
of how improvements can be made." (page 
24) . 

Assumption Four ; Evaluation has served as an effective tool for 

facilitating program improvement activities. 

Under a broadened definition of use, there is ample evidence that 
evaluation information is used by the school people to enhance their 
undei. standing ot and decisions on various educational issues (Kennedy, 
1984). Evaluation use often occurs in subtle and incremental ways 
(Alkin, et at., 1979; Patton, et al., 1977). Evaluation information is 
sorted, sifted, interpreted, and translated into implications and 
inferences (Kennedy, 1984). Use typically contributes to incremental 
decisionmaking (Wise, 1978) . 

King and Pechman (1984) identiry two principal uses of evaluati'^ns: 
signaling use and charged use. In signaling use, evaluation information 
serves as a signal from the local district to funding and legislative 
support agencies that all is well. It is done to meet accountability 
ejqpectations and requirements. In such cases, the evaluation report is 
often a routine bureaucratic statement with little or no potential for 
effecting change. Charged use, on the other hand, has the potential to 



cause a reaction xa t-he system. It provides data upon which an 
individual or a group of individuals can base decisions and actions. 

White & Worthen (1^84) demonstrated that program evaluation can 
produce valuable outcorceo in addition to complying with federal 
requirements. The authors' clients gained insights into how a state 
program designed to facilitate local innovation could be improved through 
comprehensive evaluation. Also, local district stafr gained an increased 
appreciation for the role of evaluation in imp-roving local programs under 
their wntrol. 

Alkin^ et al. , (1974) and Patton et al. , (1977) believe thc.t 
recommendations calling for massive changes in the system will be used 
less than reports that recommend minor modifications. Shapiro (1985) 
believej, that decisionmakers tend to discount judgmental infotmation 
because it is easily susceptible to strategic misrepresentation. They 
are most prone to use descriptive data rather than inferences about the 
worth or value of a program. 

In the same vein, Eash (1985) sees evaluation research moving toward 
policy-oriented research designed to provide evidence on specific 
practical problems in a tinely fashion. This calls for (a) close 
interaction between the evaluator and the client throughout the life of a 
project, (b) evolving rather than fixed evaluation dcisigns, and (c) 
increased attention to the contextual politics involved as well as 
technical demands. 

By providing frequent feedback to program staff, foraative evaluation 
he^s to modify or improve programs (Alkin, et al. , 19/4; Dickey, 1980). 
In general, formative evaluation offers much greater potential than 
summative -^luation for contributing to school or program improvement. 
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As Gottfredson (1984) implies, the pace cf organizational change often 
makes suiraaative evalution untimely. Less vigorous ways of knowing the 
effects of innovations are necessary and more useful. 

Cooley (1983) sees a need to move from formal program evaluation tc a 
client-centered systems c^proach to program improvement. He feels that 
the technological-experimental paradigm of educational change (Berman, 
1980) tends to produce results that are eithei: invalid or untimely. A 
cybernetic paradigm which entails continual "monitoring and tailoring" of 
program functions and outcomes offers considerably greater promise 
(Cooley, 1983) . 

Assumption Five ; There is a symbiotic relationship between school 

and program .improvement activities. 

A cursory review of the etfective schools literature shows that most 
researchers have used the terms "program," "curriculum," "innovation," 
"intervention," and "school" loosely. There does not seem to be a clear 
distinction between "school improvement" and "program improvement," for 
example. More recent studies have described the activities in question 
as "improvements in schooling," or "educational iinprovement." 

Most practitioners would no doubt respond with the term "curriculum" 
if asked to define the term "program." Thus, a grounded definition of 
the latter term would have us believe that "school improvement" and 
"program improvement" are conceptually and practically indistinguishable. 

To be sure, there is no empirical basis for a separation between 
school and program improvement effo' zs if "program" is defined as a set 
of curricular activities conducted ♦:o achieve some pre-specif ied 
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. "-tructional objectives. In ^ ' at sense, the day-to-day operations of 
"schooling" consist of a conpiex intermingling of schools and programs. 
Programs would not exist without schools and scnools would become empty 
structures without programs. A school is a canposite or a gestalt of 
instructional programs, A program is nested in the larger environment of 
the school, a political-administrative structure. 

Some programs, most notably federally f anded programs such as Ch^ter 
1, may enconpass more than one school. In a larger sense, however, such 
programs represent federal or state efforts to improve schools in 
providing instruction to fecial population groips such as educationally 
disadvantaged children. Improving such programs will logically lead to 
improved (derations and outcomes of schooling for all. 

Quite often, a soiiool improvement effort is focused on a particular 
program or curriculum (e.g., reading, language arts, writing) within a 
school. Thus, progreun improvement (X)ntributes directly to schc;^l 
improvement. School improvement is often operationaiized as> specific 
program improvement efforts. Although research on effective schools 
calls for a schoolwide emphasis, studies of effective schools seldom 
measure the instructional perforuicnce of an entire school. Assessment of 
instructional outcomes generally occurs at only one or two grade levels 
and in only one or two curriculum areas (Rowan, et al. , 1983). 

School improvement activities must focus on fine-grained within-school 
processes. Teachers, for example, need to understand the attributes of 
effective classroom instruction and learn how to manage different types 
of instructional systems. The "stuff" of schooling is the curricula, or 
programs, offered in a school. It is that "stuff" and factors which 
affect its effectiveness (e.g., teacher expectations, time-on- task) that 
we should attempt to improve. 
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There are, undoubtedly, some school- level factx>rs (e.g., school 
climate, rules on discipline, pep rallies) the improvement of which cuts 
across all programs within a school building. They represent, however, a 
minority of areas to which improvement activities can be directed. 

To say that there is no empirical distinction between school and 
program improvement does not. deny the fact that the school as a 
functioning social system should be the focus of change, program adopt. ion 
and effective reform. It merely reflects the reality that most 
improvement activities occur on a long-term incremental basis to fine 
tune the system: one school at a time, one program at a time (Lindblom, 
1^72; Colley, 1983) . 

7. TECHNICAL ASSISTANCE ROLE 

Commenting on his experience in providing research support to a large 
city school systemr Cooley (1983) offers two central prescriptons for 
evaluators. The first is to maintain a client orientation and the second 
is to shift emphasis from formal program evaluation to a systems approach 
to program improvement. On the latter point Cooley observes that: 

"educational research that takes place within school districts 
. . . would profit greatly if the emphasis were shifted from 
discrete studies of particular progranttJ or policies, which 
generally fall under the rubric of program evaluation, to a 
continuous activity of data collection and analysis, which I 
refer to as monitoring and tailoring." (p. 7) 
Adding a point that seems to challenge the usefulness of the Title I 
Evaluation and Reporting System (TIERS) , Cooley argues that: 
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"Formal, suinmative program evaluations that attempt to 
estimate the impact cf a particular program or policy 
on student outcomes tend to ^">roduce results that should 
not be used, because of their invalidity, or cannot be 
used, because valid impact studies, if they can be done 
at all, take too long to be tijnely. (p. 7) 

Nonetheless, Congress has mandated that local districts not only 
evaluate their Chapter 1 programs, but also use the results of such 
evaluations to improve their programs. TkC staff and evaluators of 
Chapter 1 programs obviously need to reassess and reformulate their roles 
to meet this mandate. 

In reformulating their role, TAC staff must loo;, to a broader range of 
tasks and greater flexibility in performing these tasks. Chapter 1 
evaluations must go beyond the determination of summative impact. TAC 
role might more accurately be described as ••improvement monitoring" to be 
shaped by the following considerations: 

Efficiency . Evaluation, e^ecially summative evaluation, speaks 
primarily to the issue of effectiveness. Improvement monitoring 
addresses che issue of efficiency. While same improvement efforts would 
advocate increased financial support for schools, most focus on the more 
efficient use o£ existing resources (Edmonds, 1982). As Mackenzie (1983) 
suggests, the amount of agreement on the principal factors in school 
effectiveness is so impressive that the more relevant question now is 
what can be changed for the least cost and the most results. 
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Self-^Renewal * School or program improvement is not a one-shot 
solution to organizational problems* It is an attempt to grow toward 
greater effectiveness through a series of intervention activities over a 
period of time (French & Bell, 1978) • Improvement monitoring is a 
cyclical process^ It is self-renewing as progress is made toward 
achieving improvement goals and objectives, as new research findings are 
made, as the school environment changes, and as new evaluation questions 
emerge. 

Theor y-0 r i en ta t ion . Improvement monitoring is more theory-oriented 
than evaluation as we know it at present. Through theory-guided 
inprovement monitoring, TAG staff not only assist in solving problems, 
but also learn how and why the problems are solved. They, therefore, 
need to be more of a researcher and synthesizer or research findings than 
they are at present. Theory provides a template for judging the 
appropriateness of the interventions and the objectives of an improvement 
ettort (Gottf reason, 1984). Improvement monitorinq seeks to explain the 
causal links among educational phenomena. It is not enough simply to 
detect differences in re^»lity. We need to explain vdiat accounts for the 
difference, (Kerlinger, 1977). Improvement monitoring focuses on not only 
the outcomes of schooling but also the teaching-learning process, 
especially the linkage of resource i o resource use and to student 
performance. Improvement prescriptions assume causal links are 
understood. Monitoring not only checks on whether events have taken 
place as planned, but also attempts to detect whether such events ate 
producing the e^^ected consequences. 

Data Base . In improvement monitoring, the e valuator helps improve 
programs by (a) serving as a technical resource in data gathering. 
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analysis^ interpretation and display; (b) serving as a sounding board for 
new ideas; and (c) improving the knowledge base of program inplementers. 

To generate data of maximum utility^ the evaluator must produce 
relatively neutral, descriptive data relevant to the decisionmaker. The 
evaluator needs to present the data results, rather than recommendations, 
to the decisionmaker, and permit the latter to draw his or her own 
conclusions (Shapiro, 1985). This is particularly true in such 
activities as nends assessment, standard setting, and cost analysis. 

Standard Setting . In improvement monitoring, TAG staff may offer 
assistance in setting quality as well as comparative standards. We need 
standards for judging whether student performance is satisfactory or 
unsatisfactory, acceptable or unacceptable. We also need standards for 
judging whether one group performs better or worse than another, or 
whether a group's current performance is better or worse than before 
(Messick, 1985) . TAG staff can assist in establishing a responsible 
standard setting process by attending to three key elements! (1) the 
choice of educational objectives, (2) the description of current group 
performanco ranges and trends, and (3) the identification of educational 
contexts differentially related to performance (Messick, 1985) . 

Collv^boration . TAG staff must adopt a more collaborative posture to 
involve school administrators and staff in improvement monitoring. There 
is ample evidence that evaluation work conducted without any staff 
Involvement is less likely to be valid and useful (Dickey, 1980) . Most 
ej^erts believe that the separation between evaluators and curriculum 
practitioners will become less sharp in the future and the two roles may 
be merged in most field practices (Eash, 1985). 
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VI, CXINCLUDING REMARKS 



None of the four categories of improvement activity should be viewed 
in pe3orative light • Each has its place in the overall change process. 
Symbolic improvement activities, for exanple, are symbolic only in tiie 
sense that no direct improvement takes place during the activity. As a 
dissemination tool, such activities have a significant role to play in 
raising awareness and in generating momentum and enthusiasm for setting 
the stage for greater efforts in the future. 

The present study showed that a majority of the program improvement 
activities in Chapter 1 are found in the opportunistic or pragmatic 
category. Recent field e3q>eriences suggest that this trend is likely to 
continue in the near future. The finding lends credence to the belief 
that change efforts which are actually implemented at the SEA and LEA 
levels are more likely to be small and incremental rather than extensive 

particularly true of Chapter 1 since such projects are typically a small 
part of the overall school operations. 

This does not mean, of course, that systematic appro ::hes to 
improvement she \ld be discouraged or abandoned altogether. It does 
imply, however, that understanding of and expectations for program 
improvement in Chapter 1 should be moderated in the face of reality 
constraints. It further implies that many of the school improvement 
models currently in use may need to be adapted or tailored to accommodate 
assumptions inherent in the improvement package and that TAC staff should 
take advantage of the diversity of approaches being initiated to afford 
flexibility in promoting incremental improvements in Chapter 1 projects. 
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A shared knowledge base, consensus decisionmaking, and a participatory 
and collaborative process appear to be essential ingredients of a 
successtul program improvement effort. More significantly, the pivotal 
importance of the teacher and the classroom is evident in all cases. 
Higher level processes such as state or district initiative, principal 
leadership, and e^qpertise of outside consultants are imporcant to the 
extent that they can play an effective role in facilitating changed 
behavior and improved student- teacher interaction in the classroom. 

The role of evaluation will be shaped by the client's need for 
information both to establish accounteUDility and to suH>ort improvement 
activities. Technology has increased our capacity to collect, process, 
and analyze vast amounts of data. This will provide greater choices with 
regard to the priority for different types of data. Making those choices 
may create tensions between different TAG client groqps. TAG stafl will 
need to be cognizant of the divergent needs and objectives of the 
different client qroic>s. They need to be aware of both the norinative and 
prescriptive models of decisionmaking (Kennedy, 1984), Evaluation 
information is used for develqping understanding as well as for 
supporting specific decisioiis. Our ideas of how decisions are made 
prompt us to use diiierent strategies for promoting appropriate 
evaluation use and for encouraging such use (King and Pechman, 1984) . 

Ev^lUators trained in research methods may iiold a skeptical view 
toward improvement initiatives. They are used to testing the null 
hypotheses instead of the researcft hypotheses. Nonetheless, linking 
evaluation to program improvement efforts is a natural Lnd necessary 
extension of the evaluator's work. To be effective and, indeed, to 
ensure its survival^ the evaluation profession must exhibit "a bias 
toward action" (Peters and Waterman 1982). 
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